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Abstract 

A monadic parser combinator library which guarantees termination 
of parsing, while still allowing many forms of left recursion, is 
described. The library's interface is similar to those of many other 
parser combinator libraries, with two important differences: one is 
that the interface clearly specifies which parts of the constructed 
parsers may be infinite, and which parts have to be finite, using 
dependent types and a combination of induction and coinduction; 
and the other is that the parser type is unusually informative. 

The library comes with a formal semantics, using which it is 
proved that the parser combinators are as expressive as possible. 
The implementation is supported by a machine-checked correct- 
ness proof. 

Categories and Subject Descriptors D.l.l [Programming Tech- 
niques]: Applicative (Functional) Programming; E. 1 [Data Struc- 
tures]; F.3.1 [Logics and Meanings of Programs]: Specifying and 
Verifying and Reasoning about Programs; F.4.2 [Mathematical 
Logic and Formal Languages]: Grammars and Other Rewriting 
Systems — Grammar types, Parsing 

General Terms Languages, theory, verification 

Keywords Dependent types, mixed induction and coinduction, 
parser combinators, productivity, termination 

1. Introduction 

Parser combinators (Burge 1975; Wadler 1985; Fairbairn 1987; 
Hutton 1992; Meijer 1992; Fokker 1995; Rojemo 1995; Swier- 
stra and Duponcheel 1996; Koopman and Plasmeijer 1999; Lei- 
jen and Meijer 2001; Ljunglof 2002; Hughes and Swierstra 2003; 
Claessen 2004; Frost et al. 2008; Wallace 2008, and many others) 
can provide an elegant and declarative method for implementing 
parsers. When compared with typical parser generators they have 
some advantages: it is easy to abstract over recurring grammatical 
patterns, and there is no need to use a separate tool just to parse 
something. On the other hand there are also some disadvantages: 
there is a risk of lack of efficiency, and parser generators can give 
static guarantees about termination and non-ambiguity which most 
parser combinator libraries fail to give. This paper addresses one of 
these points by defining a parser combinator library which ensures 
statically that parsing will terminate for every finite input string. 
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The library has an interface which is very similar to those of 
classical monadic parser combinator libraries. For instance, con- 
sider the following simple, left recursive, expression grammar: 

term ::— factor \ term ' + ' factor 
factor ::= atom \ factor '*' atom 
atom ::— number \ '{'term')' 

We can define a parser which accepts strings from this grammar, 
and also computes the values of the resulting expressions, as fol- 
lows (the combinators are described in Section 4): 

mutual 

term — factor 

| tt term »= X n\ — » 

tok ' + ' X _ 

factor 3= X «2 — > 

return (n\ + n^) 

factor — atom 

| ^factor »= X n\ — > 

tok'*' X _ -> 

atom »= X «2 — > 

return (rci * n-i) 

atom = number 

\ tok' V »= X _ 

tt term 3= In -> 

tok')' X _ -» 

return n 

The only visible difference to classical parser combinators is the 
use of which indicates that the definitions are corecursive (see 
Section 2). However, we will see later that the parsers' types con- 
tain more information than usual. 

When using parser combinators the parsers/grammars are often 
constructed using cyclic definitions, as above, so it is natural to see 
the definitions as being partly corecursive. However, a purely coin- 
ductive reading of the choice and sequencing combinators would 
allow definitions like the following ones: 

P 

p' — tt p' 3= A jc — » f return (fx) 

For these definitions it is impossible to implement parsing in a total 
way (in the absence of hidden information): a defining characteris- 
tic of parser combinator libraries is that non-terminals are implicit, 
encoded using the recursion mechanism of the host language, so (in 
a pure setting) the only way to inspect p and p' is via their infinite 
unfoldings. The key idea of this paper is that, even if non-terminals 
are implicit, totality can be ensured by reading choice inductively, 
and only reading an argument of the sequencing operator coinduc- 
tively if the other argument does not accept the empty string (see 
Section 3). To support this idea the parsers' types will contain in- 
formation about whether or not they accept the empty string. 



The main contributions of the paper are as follows: 

• It is shown how parser combinators can be implemented in such 
a way that termination is guaranteed, using a combination of 
induction and coinduction to represent parsers, and a variant of 
Brzozowski derivatives (1964) to run them. 

• Unlike many other parser combinator libraries these parser 
combinators can handle many forms of left recursion. 

• The parser combinators come with a formal semantics. The 
implementation is proved to be correct, and the combinators 
are shown to satisfy a number of laws. 

• It is shown that the parser combinators are as expressive as 
possible (see Sections 3.5 and 4.5). 

The core of the paper is Sections 3 and 4. The former section 
introduces the ideas by using recognisers (parsers which do not 
return any result other than "the string matched" or "the string 
did not match"), and the latter section generalises to full parser 
combinators. Related work is discussed below. 

As mentioned above the parser type is defined using mixed in- 
duction and coinduction (Park 1980). This technique is explained 
in Section 2, and discussed further in the conclusions. Those read- 
ers who are not particularly interested in parser combinators may 
still find the paper useful as an example of the use of this technique. 

The parser combinator library is defined in the dependently 
typed functional programming language Agda (Norell 2007; Agda 
Team 2010), which will be introduced as we go along. The library 
comes with a machine-checked 1 proof which shows that the imple- 
mentation is correct with respect to the semantics. The code which 
the paper is based on is at the time of writing available from the 
author's web page. 

1.1 Related work 

There does not seem to be much prior work on formally veri- 
fied termination for parser combinators (or other general parsing 
frameworks). McBride and McKinna (2002) define grammars in- 
ductively, and use types to ensure that a token is consumed before a 
non-terminal can be encountered, thereby ruling out left recursion 
and non-termination. Danielsson and Norell (2008) and Koprowski 
and Binsztok (2010) use similar ideas; Koprowski and Binsztok 
also prove full correctness. Muad'Dib (2009) uses a monad anno- 
tated with Hoare-style pre- and post-conditions (Swierstra 2009) 
to define total parser combinators, including a fixpoint combina- 
tor whose type rules out left recursion by requiring the input to be 
shorter in recursive calls. Note that none of these other approaches 
can handle left recursion. The library defined in this paper seems 
to be the first one which both handles (many forms of) left recur- 
sion and guarantees termination for every parser which is accepted 
by the host language. 2 It also seems fair to say that, when com- 
pared to the other approaches above, this library has an interface 
which is closer to those of "classical" parser combinator libraries. 
In the classical approach the ordinary general recursion of the host 
language is used to implement cyclic grammars; this library uses 
"ordinary" corecursion (restricted by types, see Section 3). 

There are a number of parser combinator libraries which can 
handle various forms of left recursion, but they all seem to come 



Note that the meta-theory of Agda has not been properly formalised, and 
Agda's type checker has not been proved to be bug-free, so take words such 
as "machine-checked" with a grain of salt. 

2 Danielsson and Norell (2009) define a parser using a specialised version 
of the library described in this paper. This version of the library can handle 
neither left nor right recursion, and is restricted to parsers which do not ac- 
cept the empty string. A brief description of the parser interface is provided, 
but the implementation of the backend is not discussed. 



with some form of restriction. The combinators defined here can 
handle many left recursive grammars, but not all; for instance, 
the definition p — p is rejected statically. Lickman (1995) de- 
fines a library which can handle left recursion if a tailor-made 
fixpoint combinator, based on an idea due to Philip Wadler, is 
used. He proves (informally) that parsers defined using his com- 
binators are terminating, as long as they are used in the right 
way; the argument to the fixpoint combinator must satisfy a non- 
trivial semantic criterion, which is not checked statically. Johnson 
(1995) and Frost et al. (2008) define libraries of recogniser and 
parser combinators, respectively, including memoisation combina- 
tors which can be used to handle left recursion. As presented these 
libraries can fail to terminate if used with grammars with an in- 
finite number of non-terminals — for instance, consider the gram- 
mar { p n ::— | n e N }, implemented by the definition p n — 
memoise n (p (1 + n)) — and users of the libraries need to en- 
sure manually that the combinators are used in the right way. The 
same limitations apply to a library described by Ljunglof (2002). 
This library uses an impure feature, observable sharing (Claessen 
and Sands 1999), to detect cycles in the grammar. Claessen (2001) 
mentions a similar implementation, attributing the idea to Magnus 
Carlsson. Kiselyov (2009) also presents a combinator library which 
can handle left recursion. Users of the library are required to anno- 
tate left recursive grammars with something resembling a coinduc- 
tive delay constructor. If this constructor is used incorrectly, then 
parsing can terminate with the wrong answer. 

Baars et al. (2009) represent context-free grammars, including 
semantic actions, in a well-typed way. In order to avoid problems 
with left recursion when generating top-down parsers from the 
grammars they implement a left-corner transformation. Neither 
correctness of the transformation nor termination of the generated 
parsers is proved formally. Brink et al. (2010) perform a similar 
exercise, giving a partial proof of correctness, but no proof of 
termination. 

In Section 4.5 it is shown that the parser combinators are as 
expressive as possible — every parser which can be implemented 
using the host language can also be implemented using the com- 
binators. In the case of finite token sets this holds even for non- 
monadic parser combinators using the applicative functor inter- 
face (McBride and Paterson 2008); see Section 3.5. The fact that 
monadic parser combinators can be as expressive as possible has al- 
ready been pointed out by Ljunglof (2002), who also mentions that 
applicative combinators can be used to parse some languages which 
are not context-free, because one can construct infinite grammars 
by using parametrised parsers. It has also been known for a long 
time that an infinite grammar can represent any language, decidable 
or not (Solomon 1977), and that the languages generated by many 
infinite grammars can be decided (Mazurkiewicz 1969). However, 
the result that monadic and applicative combinators have the same 
expressive strength for finite token sets seems to be largely un- 
known. For instance, Claessen (2004, page 742) claims that "with 
the weaker sequencing, it is only possible to describe context-free 
grammars in these systems". 

Bonsangue et al. (2009, Example 2) represent a kind of regular 
expressions in a way which bears some similarity to the representa- 
tion of recognisers in Section 3. Unlike the definition in this paper 
their definition is inductive, with an explicit representation of cy- 
cles: fix.e, where e can contain x. However, occurrences of x in e 
have to be guarded by what amounts to the consumption of a token, 
just as in this paper. 

In Sections 3.3 and 4.2 Brzozowski derivative operators (Brzo- 
zowski 1964) are implemented for recognisers and parsers, and in 
Sections 3.4 and 4.3 these operators are used to characterise recog- 
niser and parser equivalence coinductively. Rutten (1998) performs 
similar tasks for regular expressions. 



2. Induction and coinduction 

The parser combinators defined in Sections 3 and 4 use a combina- 
tion of induction and coinduction which may at first sight seem be- 
wildering, so let us begin by discussing induction and coinduction. 
This discussion is rather informal. For more theoretical accounts of 
induction and coinduction see, for instance, the works of Hagino 
(1987) and Mendler (1988). 

Induction can be used to define types where the elements have 
finite "depth". A simple example is the type of finite lists. In 
Agda this data type can be defined by giving the types of all the 
constructors: 

data List (A : Set) : Set where 
[ ] : List A 

_::_ : A — > List A — > List A 

This definition should be read inductively, i.e. all lists have finite 
length. Functions with underscore in their names are operators; _ 
marks the argument positions. For instance, the constructor is 
an infix operator. Set is a type of small types. 

Coinduction can be used to define types where some elements 
have infinite depth. Consider the type of potentially infinite lists 
(colists), for instance: 

data Colist (A : Set) : Set where 
[ ] : Colist A 

_::_ : A oo (Colist A) Colist A 

(Note that constructors can be overloaded.) The type function 
oo : Set — > Set marks its argument as being coinductive. It is simi- 
lar to the suspension type constructors which are used to implement 
non-strictness in strict languages (Wadler et al. 1998). Just as the 
suspension type constructors the function oo comes with delay and 
force functions, here called L (sharp) and ^ (flat): 

L : {A : Set} -> A -> oo A 
7 : [A : Set] — > oo A — > A 

Sharp is a tightly binding prefix operator; ordinary function appli- 
cation binds tighter, though. (Flat is an ordinary function.) Note 
that [A : Set} — > T is a dependent function space; the argument A 
is in scope in T. Arguments in braces, {...}, are implicit, and do not 
need to be given explicitly as long as Agda can infer them from the 
context. 

Agda is a total language. This means that all computations of 
inductive type must be terminating, and that all computations of 
coinductive type must be productive. A computation is productive 
if the computation of the next constructor is always terminating, so 
even though an infinite colist cannot be computed in finite time we 
know that the computation of any finite prefix has to be terminating. 
For types which are partly inductive and partly coinductive the 
inductive parts must always be computable in finite time, while the 
coinductive parts must always be productively computable. 

To ensure termination and productivity Agda employs two basic 
means for defining functions: inductive values can be destructed us- 
ing structural recursion, and coinductive values can be constructed 
using guarded corecursion (Coquand 1994). As an example of the 
latter, consider the following definition of map for colists: 

map : V {A B) — > (A — > B) — > Colist A Colist B 

mapf [] = [] 

map f (x :: xs) = fx :: " mapf ( v xs) 

(Note that the code V|AS)-> ... means that the function takes 
two implicit arguments A and B; it is not an application of A to B.) 
Agda accepts this definition because the corecursive call to map is 
guarded: it occurs under the delay constructor without any non- 
constructor function application between the left-hand side and the 



corecursive call. It is easy to convince oneself that, if the input colist 
is productively computable, then the (spine of the) output colist 
must also be. 

Let us now consider what happens if a definition uses both 
induction and coinduction. We can define a language of "stream 
processors" (Carlsson and Hallgren 1998; Hancock et al. 2009), 
taking colists of As to colists of Bs, as follows: 

data SP (AS : Set) : Set where 
get : (A^ SP A B) SP A B 

put : B ^ oo (SPA B) ^ SPA B 
done : SP A B 

The recursive argument of get is inductive, while the recursive ar- 
gument of put is coinductive. The type should be read as the nested 
fixpoint vX.ji Y. (A — » ¥) + B x X + 1 , with an outer greatest fix- 
point and an inner least fixpoint. 3 This means that a stream proces- 
sor can only read (get) a finite number of elements from the input 
before having to produce (put) some output or terminate (done). 
As a simple example of a stream processor, consider copy, which 
copies its input to its output: 

copy : V {A} -> SPAA 

copy = get (la-> put a ($ copy)) 

Note that copy is guarded (lambdas do not affect guardedness). 
The semantics of stream processors can be defined as follows: 

[_] : V {A B} SP A B Colist A Colist B 
I get/ ] (a:: as) = if a] (^ as) 
I put b sp }as = b:-J I 7 sp ] as 

I- 1- = [] 

([_] is a mixfix operator.) In the case of get one element from the 
input colist is consumed (if possible), and potentially used to guide 
the rest of the computation, while in the case of put one output 
element is produced. The definition of [_] uses a lexicographic 
combination of guarded corecursion and structural recursion: 

• In the second clause the corecursive call is guarded. 

• In the first clause the corecursive call is not guarded, but it 
"preserves guardedness": it takes place under zero occurrences 
of tt_ rather than at least one (and there are no destructors in- 
volved). Furthermore the stream processor argument is struc- 
turally smaller: / x is strictly smaller than get / for any x. 

This ensures the productivity of the resulting colist: the next output 
element can always be computed in finite time, because the number 
of get constructors between any two put constructors must be 
finite. Agda accepts definitions which use this kind of lexicographic 
combination of guarded corecursion and structural recursion. For 
more information about Agda's criterion for accepting a program 
as total, and more examples of the use of mixed induction and 
coinduction in Agda, see Danielsson and Altenkirch (2010). 

It may be interesting to observe what would happen if get 
were made coinductive. In this case we could define more stream 
processors, for instance the following one: 

sink : V (AS)-> SP A B 
sink = get (X _ — > f sink) 

On the other hand we could no longer define [_] as above (suitably 
modified), because the output of [ sink ] as would not be produc- 
tive for infinite colists as. In other words, if we make more stream 
processors definable some functions become impossible to define. 



3 At the time of writing this interpretation is not correct in Agda (Altenkirch 
and Danielsson 2010), but the differences are irrelevant for this paper. 



3. Recognisers 

This section defines a small embedded language of parser combi- 
nators. To simplify the explanation the parser combinators defined 
in this section can only handle recognition. Full parser combinators 
are described in Section 4. 

The aim is to define a data type with (at least) the following 
basic combinators as constructors: fail, which always fails; empty, 
which accepts the empty string; sat, which accepts tokens satisfy- 
ing a given predicate; _|_, symmetric choice; and sequencing. 

Let us first consider whether the combinator arguments should 
be read inductively or coinductively. An infinite choice cannot 
be decided (in the absence of extra information), as this is not 
possible without inspecting every alternative, so choices will be 
read inductively. The situation is a bit trickier for sequencing. 
Consider definitions like p = p ■ p' or p = p' ■ p. If p' accepts the 
empty string, then it seems hard to make any progress with these 
definitions. However, if p' is guaranteed not to accept the empty 
string, then we know that any string accepted by the recursive 
occurrence of p has to be shorter than the one accepted by p ■ p' 
or p' ■ p. To make use of this observation I will indicate whether 
or not a recogniser is nullable (accepts the empty string) in its type, 
and the left (right) argument of _•_ will be coinductive iff the right 
(left) argument is not nullable. 

Based on the observations above the type P of parsers (recog- 
nisers) can now be defined for a given token type Tot. 

mutual 

data P : Bool -> Set where 
fail : P false 
empty : P true 

sat : (Tok -> Bool) -> P false 
_|_ : V {«i n 2 ) -> P n\ -> P n 2 -> P in\ v n 2 ) 
: V {«! n 2 ] -> 
oo( n 2 )P n\ — > oo( n\ )P n 2 — > P (n\ A n 2 ) 

oo(_)P : Bool -> Bool -> Set 
oo( false )P n = oo (P ri) 
oo( true )P n = P n 

Here P true represents those recognisers which accept the empty 
string, and P false those which do not: fail and sat do not accept 
the empty string, while empty does; a choice p\ \ p 2 is nullable 
if either p\ or p 2 is; and a sequence pi ■ p 2 is nullable if both p\ 
and p 2 are. The definition of the sequencing operator makes use of 
the mixfix operator co(_)P to express the "conditional coinduction" 
discussed above: the left argument has type oo( n 2 )P n\, which 
means that it is coinductive iff n 2 is false, i.e. iff the right argument 
is not nullable. The right argument's type is symmetric. 

The conditionally coinductive type oo{_)P comes with corre- 
sponding conditional delay and force functions: 

tt ? : V {b n) -> P n -> oo( b )P n 
tt ? {b = false} x = tt x 
J- [b — true} x — x 

t> ? : V [b n) -> oo( b )P n ^ P n 
\r [b = false} x — ^ x 
b" [b — true} x — x 

(Here [b = . . .} is the notation for pattern matching on an implicit 
argument.) We can also define a function which returns true iff the 
argument is already forced: 

forced! : V [b n\ -> oo( b )P n -> Bool 
forced! [b = b] _ = b 

In addition to the constructors listed above the following con- 
structors are also included in P: 



nonempty : V {«} — > P n — > P false 

cast : V (wj n 2 ) — > ri[ = n 2 — > P ri[ — > P n 2 

The nonempty combinator turns a recogniser which potentially 
accepts the empty string into one which definitely does not (see 
Section 3.1 for an example and 3.2 for its semantics), and cast 
can be used to coerce a recogniser indexed by n\ into a recogniser 
indexed by n 2 , assuming that n\ is equal to n 2 (the type n\ = n 2 is 
a type of proofs showing that n\ and n 2 are equal). Both nonempty 
and cast are definable in terms of the other combinators — in the 
case of cast the definition is trivial, and nonempty can be defined 
by recursion over the inductive structure of its input — but due to 
Agda's reliance on guarded corecursion it is convenient to have 
them available as constructors. 

3.1 Examples 

Using the definition above it is easy to define recognisers which are 
both left and right recursive, for instance the following one: 

left-right : P false 

left-right — " left-right ■ f left-right 

Given the semantics in Section 3.2 it is easy to show that left-right 
does not accept any string. This means that fail does not necessarily 
have to be primitive, it could be replaced by left-right. 

As examples of ill-defined recognisers, consider bad and bad 2 : 

bad : P false bad 2 : P true 

bad = bad bad 2 = bad 2 ■ bad 2 

These definitions are rejected by Agda, because they are neither 
structurally recursive nor guarded. They are not terminating, either: 
an attempt to evaluate the inductive parts of bad or bad 2 would lead 
to non-termination, because the definitions do not make use of the 
delay operator L. 

As a more useful example of how the combinators above can be 
used to define derived recognisers, consider the following definition 
of the Kleene star: 

mutual 

_* : P false — > P true 
p * = empty | p + 

_+ : P false -> P false 

p + = p ■ tt (p *) 

(The combinator _|_ binds weaker than the other combinators.) 
The recogniser p ★ accepts zero or more occurrences of whatever p 
accepts, and p + accepts one or more occurrences; this is easy to 
prove using the semantics in Section 3.2. Note that this definition 
is guarded, and hence productive. 4 Note also that p must not accept 
the empty string, because if it did, then the right hand side of p + 
would have to be written p ■ p *, which would make the definition 
unguarded and non-terminating — if p * were unfolded, then no 
delay operator would ever be encountered. By using the nonempty 
combinator one can define a variant of _* which accepts arbitrary 
argument recognisers: 

:VW->P»^P true 
p -k = nonempty p-k 

For more examples, see Section 4.6. 



4 The call to p + is not guarded in the definition of p *, but all that matters 
for guardedness is calls from one function to itself. If p + is inlined it is 
clear that p * is guarded. 



3.2 Semantics 

The semantics of the recognisers is defined as an inductive family. 
The type s e p is inhabited iff the token string s is a member of the 
language defined by p: 

data _e_ : V [n] -> List Tok -> P n Set where 

The semantics is determined by the constructors of _e_, which are 
introduced below. The values of type s e p are proofs of language 
membership; the constructors can be seen as inference rules. To 
avoid clutter the declarations of bound variables are omitted in the 
constructors' type signatures. 

No string is a member of the language defined by fail, so there 
is no constructor for it in _e_. The empty string is recognised by 
empty: 

empty : [] e empty 

(Recall that constructors can be overloaded.) The singleton [ t ] is 
recognised by sat/ iff t evaluates to true (T b is inhabited iff b is 
true): 

sat : T(ft) -> [/] e sat/ 

If s is recognised by p\, then it is also recognised by p\ \ p 2 , and 
similarly for p2 : 

I - 1 eft : s e pi s e p\ \ p 2 
|-right : s e p 2 -» s e p\\p2 

If s\ is recognised by p\ (suitably forced), and s 2 is recognised 
by p2 (suitably forced), then the concatenation of s\ and S2 is 
recognised by p\ ■ p 2 : 

_•_ : si e t> ? pi s 2 e \y p 2 -» 
*1 * *2 e P1-P2 

If a nonempty string is recognised by p, then it is also recog- 
nised by nonempty p (and empty strings are never recognised by 
nonempty p): 

nonempty : t :: s e p — > t :: s e nonempty p 

Finally cast preserves the semantics of its recogniser argument: 

cast : s e p — > s e cast eq p 

It is easy to show that the semantics and the nullability index 
agree: if p : P n, then [] e p iff n is equal to true (one direction 
can be proved by induction on the structure of the semantics, and 
the other by induction on the inductive structure of the recogniser; 
delayed sub-parsers do not need to be forced). Given this result it 
is easy to decide whether or not [ ] e p; it suffices to inspect the 
index: 

nullable! : V {n} (p : P n) Dec ([ ] e p) 

Note that the correctness of nullable! is stated in its type. An 
element of Dec P is either a proof of P or a proof showing that 
P is impossible: 

data Dec (P : Set) : Set where 
yes : P Dec P 
no : -■ P -> Dec P 

Here logical negation is represented as a function into the empty 
type: -■ P = P -> ±. 

3.3 Backend 

Let us now consider how the relation _e_ can be decided, or al- 
ternatively, how the language of recognisers can be interpreted. No 
attempt is made to make this recogniser backend efficient, the focus 
is on correctness. (Efficiency is discussed further in Section 4.2.) 



The backend will be implemented using so-called derivatives 
(Brzozowski 1964). The derivative D t p of p with respect to t is the 
"remainder" of p after p has matched the token t; it should satisfy 
the equivalence 

s e D tp <:=> t :: s e p. 

By applying the derivative operator D to t\ and p, then to ?2 and 
D t\ p, and so on for every element of the input string s, one can 
decide if s e p is inhabited. 

The new recogniser constructed by D may not have the same 
nullability index as the original one, so D has the following type 
signature: 

D : V {«} (t : Tok) (p : P n) -> P (D-nullable t p) 

The function D-nullable decides whether the derivative accepts 
the empty string or not. Its extensional behaviour is uniquely con- 
strained by the definition of D; its definition is included in Figure 1. 

The derivative operator is implemented as follows. The combi- 
nators fail and empty never accept any token, so they both have the 
derivative fail: 

Or fail = fail 
D t empty = fail 

The combinator sat / has a non-zero derivative with respect to t iff 
/ 1 is true: 

D t (sat/) with/ t 
... | true = empty 
... | false = fail 

(Here the with construct is used to pattern match on the result of 
/ t.) The derivative of a choice is the choice of the derivatives of its 
arguments: 

Dt( Pl \p 2 ) = Dtpy \ Dtp2 

The derivatives of nonempty p and cast eq p are equal to the 
derivative of p: 

D t (nonempty p) — Dtp 
D t (cast eq p) —Dtp 

The final and most interesting case is sequencing: 

D t (pi ■ P2) with forced! p\ \ forced! p 2 
... | true | false = Dt p\ ■ f C Pi) 
... | false | false = tt D t ( b Pl ) ■ f ? p 2 ) 
... | true | true = D t p\ ■ jf Pl \ D t p 2 
... | false | true = 1 D t Pi ) ■ f p 2 I D t p 2 
Here we have four cases, depending on the indices of pi and p2' 

• In the first two cases the right argument is not forced, which 
implies (given the type of _•_) that the left argument is not 
nullable. This means that the first token accepted by p\ ■ p2 (if 
any) has to be accepted by pi, so the remainder after accepting 
this token is the remainder of pi followed by p 2 . 

• In the last two cases p\ is nullable, which means that the first 
token could also be accepted by P2- This is reflected in the 
presence of an extra choice D t p2 on the right-hand side. 

In all four cases the operator f is used to conditionally delay 
P2, depending on the nullability index of the derivative of p\; the 
implicit argument b to f is inferred automatically. 

The derivative operator D is total: it is implemented using a 
lexicographic combination of guarded corecursion and structural 
recursion (as in Section 2). Note that in the first two sequencing 
cases p2 is delayed, but D is not applied recursively to ^ P2 because 
Pl is known not to accept the empty string. 



D-nullable : V {n} — > Tok — > P n — > Bool 



D-nullable /fail 
D-nullable t empty 
D-nullable t (sat/) 
D-nullable t (pi | P2) 



= false 
= false 

= ft 

= D-nullable t p\ V 

D-nullable t p2 
= D-nullable t p 
= D-nullable t p 
with forced? p\ \ forced"! p2 



D-nullable t (nonempty p) 
D-nullable t (cast _ p) 
D-nullable t (p\ ■ p2) 

... | true | false = D-nullable t pi 

... | false | false = false 

... | true | true = D-nullable t p\ V D-nullable t p2 

... | false | true = D-nullable t P2 

Figure 1. The index function D-nullable. 

The index function D-nullable uses recursion on the inductive 
structure of the recogniser. Note that D-nullable does not force any 
delayed recogniser (it does not use '"). Readers familiar with de- 
pendent types may find it interesting that this definition relies on 
the fact that _A_ is defined by pattern matching on its right ar- 
gument. If _A_ were defined by pattern matching on its left argu- 
ment, then the type checker would no longer reduce the open term 
D-nullable (^ p\) t A false to false when checking the definition of 
D. This problem could be fixed by using an equality proof in the 
definition of D, though. 

It is straightforward to show that the derivative operator D 
satisfies both directions of its specification: 

D-sound : V [n s t] \p : P n] ^ s e D t p — > t :: s e p 
D-complete : V [n s t} [p : P n} — > t :: s e p — > s e Dtp 

These statements can be proved by induction on the structure of the 
semantics. 

Once the derivative operator is defined and proved correct it is 
easy to decide if s e p is inhabited: 

_e?_ : V {n} (s : List Tok) (p : P n) Dec (s e p) 

[ ] e? p = nullablel p 

t :: s e? p with .v e? Dtp 

... | yes seDtp = yes (D-sound seDtp) 

... | no s^Dtp — no (s^Dtp o D-complete) 

In the case of the empty string the nullability index tells us whether 
the string should be accepted or not, and otherwise _e?_ is recur- 
sively applied to the derivative and the tail of the string; the specifi- 
cation of D ensures that this is correct. (Note that seDtp and s^Dtp 
are normal variables with descriptive names.) 

As an aside, note that the proof returned by _e?_ when a string 
matches is actually a parse tree, so it would not be entirely incorrect 
to call these recognisers parsers. However, in the case of ambiguous 
grammars at most one parse tree is returned. The implementation 
of parse in Section 4.2 returns all possible results. 

3.4 Laws 

Given the semantics above it is easy to prove that the combinators 
satisfy various laws. Let us first define that two recognisers are 
equivalent when they accept the same strings: 

: V [n\ «2) — » P n l — » P n 2 — » Set 
PI « P2 = PI < P2 x p 2 < pi 
Here _x_ is conjunction, and encodes language inclusion: 

: V {«i «2) ~~ ► P n l ~ > P n 2 ~ > Set 
Pi < P2 = v M -> s e Pi -» s e P2 



It is straightforward to show that is an equivalence relation, 
if the definition of "equivalence relation" is generalised to accept 
indexed sets. (Such generalisations are silently assumed in the 
remainder of this text.) It is also easy to show that is a 
congruence — i.e. that it is preserved by all the primitive recogniser 
combinators — and that is a partial order with respect to 

The following definition provides an alternative, coinductive 
characterisation of equality: 

data _^ c - {«i "2} (pi '■ P n\) (p2 ■ P "2) : Set where 

_::_ : nj = n 2 -> (V t -> 00 (D tp\ ^ c Dt P2)) -* 
PI P2 

Two recognisers are equal iff they agree on whether the empty 
string is accepted, and for every token the respective derivatives 
are equal (coinductively). Note that the values of this data type are 
infinite proofs 5 witnessing the equivalence of the two parsers. Note 
also that this equality is a form of bisimilarity: the "transitions" are 
of the form 

p — ^> Dtp, 

where p : P n. It is easy to show that and _^ c - are equivalent. 
When proving properties of recognisers one can choose the equality 
which is most convenient for the task at hand. For an example of a 
proof using a coinductively defined equality, see Section 4.4. 

The type of the sequencing combinator is not quite right if we 
want to state properties such as associativity, so let us introduce the 
following variant of it: 

_©_ : V {n\ n 2 } -> P n\ -> P n 2 -» P (n\ A n 2 ) 

-O-W = rail Pi P2 = f Pi ■ f {b = «l) P2 

(Agda does not manage to infer the value of the implicit argument 
b, but we can still give it manually.) Using the combinator _©_ it 
is easy to prove that the recognisers form an idempotent semiring: 



P\P 
fail I p 
p O empty 
empty © p 
fail © p 
p © fail 



P Pl\P2 

P Pi I (P2 I P3) 

P PI © (P2 © P3) 

P Pi © (P2 I P3) 

fail (pi \p 2 )QP3 

fail 



P2 I Pi 
(Pi \P2) \P3 
(pi QP2) OP3 
Pi OP2\PlQP3 
Pi © P3 I P2 © P3 



It is also easy to show that the order coincides with the natural 
order of the join-semilattice formed by _|_: 



Pi < P2 <=> Pl\ P2 



P2 



By using the generalised Kleene star from Section 3.1 one 
can also show that the recognisers form a -^-continuous Kleene 
algebra (Kozen 1990): p\ 0 (p2 *) 0 P3 is the least upper 
bound of the set {pj 0 (p2 " i) O P3 \ 1 : e N }, where p " i is the 
i-fold repetition of p: 

{_"_)-nullable : Bool -> N -» Bool 
(n" zero )-nullable — _ 
( n " sue i )-nullable — _ 

_"_ : V {«} P n -> (i : N) ^ P (( n " i )-nullable) 
p " zero = empty 
p " sue = p 0 (p " i) 

(Here zero and sue are the two constructors of N. Note that Agda 
can figure out the right-hand sides of (_'_)-nullable automatically, 
given the definition of see Section 4.6.) 

3.5 Expressive strength 

Is the language of recognisers defined above useful? It may not 
be entirely obvious that the restrictions imposed to ensure totality 



' If Tok is non-empty. 



do not rule out the definition of many useful recognisers. Fortu- 
nately this is not the case, at least not if Tok, the set of tokens, is 
finite, because then it can be proved that every function of type 
List Tok — > Bool which can be implemented in Agda can also 
be realised as a recogniser. For simplicity this will only be shown 
in the case when Tok is Bool. The basic idea is to turn a function 
/ : List Bool —> Bool into a grammar representing an infinite 
binary tree, with one node for every possible input string, and to 
make a given node accepting iff / returns true for the correspond- 
ing string. 

Let us first define a recogniser which only accepts the empty 
string, and only if its argument is true: 

accept-if-true : V b — > P b 
accept-if-true true = empty 
accept-if-true false = fail 

Using this recogniser we can construct the "infinite binary tree" 
using guarded corecursion: 

grammar : (f : List Bool — > Bool) — > P (f [ ]) 
grammar/ — cast (lemma f) ( 

f (sat id) ■ * grammar (f o true ) 

| f (sat not) ■ f grammar if o _::_ false) 

| accept-if-true (f [ ])) 

Note that sat id recognises true, and sat not recognises false. The 
following lemma is also used above: 

lemma : 

V/-> (false Af [true] v false a/ [false ]) v/ [] =/ [] 

The final step is to show that, for any string s, f s = true iff 
s e grammar/. The "only if" part can be proved by induction on 
the structure of s, and the "if" part by induction on the structure of 
£ e grammar f. 

Note that the infinite grammar above has a very simple struc- 
ture: it is LL(1). I suspect that this grammar can be implemented 
using a number of different parser combinator libraries. 

As an aside it may be interesting to know that the proof above 
does not require the use of lemma. The following left recursive 
grammar can also be used: 

grammar : (f : List Bool — > Bool) — > P (f [ ]) 
grammar f = 

" grammar (X xs — > / (xs 4f [ true ])) • f (sat id) 

| tl grammar (X xs — > / (xs -H- [ false ])) • f (sat not) 

| accept-if-true (f [ ]) 

This shows that nonempty and cast are not necessary to achieve 
full expressive strength, because neither grammar nor the backend 
rely on these operators. 

Finally let us consider the case of infinite token sets. If the 
set of tokens is the natural numbers, then it is quite easy to see 
that it is impossible to implement a recogniser for the language 
[nn | n e N}. By generalising the statement to "it is impossible 
that p accepts infinitely many identical pairs, and only identical 
pairs and/or the empty string" (where an identical pair is a string 
of the form nn) one can prove this formally by induction on the 
structure of p (see the accompanying code). Note that this restric- 
tion does not apply to the monadic combinators introduced in the 
next section, which have maximal expressive strength also for infi- 
nite token sets. 

4. Parsers 

This section describes how the recogniser language above can be 
extended to actual parser combinators, which return results. 



Consider the monadic parser combinator bind, _»=_: The 
parser py »»= pj successfully returns a value y for a given string 
s if pi parses a prefix of s, returning a value x, and p2 x parses 
the rest of s, returning y. Note that py »= pi accepts the empty 
string iff py accepts the empty string, returning a value x, and pi x 
also accepts the empty string. This shows that the values which a 
parser can return without consuming any input can be relevant for 
determining if another parser is mailable. 

This suggests that, in analogy with the treatment of recognisers, 
a parser should be indexed by its "initial set" — the set of values 
which can be returned when the input is empty. However, some- 
times it is useful to distinguish two grammars if the number of 
parse trees corresponding to a certain string differ. For instance, 
the parser backend defined in Section 4.2 returns twice as many re- 
sults for the parser p \ p as for the parser p. In order to take account 
of this distinction parsers are indexed by their return types and their 
"initial bags" (or multisets), represented as lists: 

mutual 

data Parser : (R : Set) — > List R — > Sety where 

(Sety is a type of large types; Agda is predicative.) 

The first four combinators have relatively simple types. The 
return combinator is the parser analogue of empty. When accept- 
ing the empty string it returns its argument: 

return : V [R] (x : R) -> Parser R[x] 

(Note that [_] is the return function of the list monad.) The fail 
parser, which mirrors the fail recogniser, always fails: 

fail : V {R} -> Parser R [] 

(Note that [ ] is the zero of the list monad.) The token parser accepts 
any single token, and returns this token: 

token : Parser Tok [ ] 

This combinator is not as general as sat, but a derived combinator 
sat is easy to define using token and bind, see Section 4.6. The 
analogue of the choice recogniser is _|_: 

_|_ : V {R xsy xsj] — > Parser R xs\ — > Parser R XS2 — > 
Parser R (xsy -H- xs^) 

The initial bag of a choice is the union of the initial bags of its two 
arguments. 

The bind combinator's type is more complicated than the types 
above. Consider py »= p2 again. Here p2 is a function, and we 
have a function f : Ry — » List R2 which computes the initial 
bag of p2 x, depending on the value of x. When should we allow 
pi to be coinductive? One option is to only allow this when/x 
is empty for every x, but I do not want to require the user of the 
library to prove such a property just to define a parser. Instead I 
have chosen to represent the function / with an optional function 
/ : Maybe (Ry —> List R2), 6 where nothing represents X _ — > [ ], 
and to make p\ coinductive iff / is nothing. The same approach is 
used for xs, the initial bag of p\ : 

_»=_ : V {Ry R 2 \ {xs : Maybe (List Ry)} 
\f : Maybe (Ry -> List R 2 )} -> 
oo(/ )Parser Ry (flatten xs) — > 
((x : Ry) — > oo( xs )Parser R2 (apply f x)) — > 
Parser R2 (bind xs f) 

The helper functions flatten, apply and bind, which interpret 
nothing as the empty list or the constant function returning the 



The type Maybe A has the two constructors nothing : Maybe A and 
just : A -» Maybe A. 



flatten : [A : Set) — > Maybe {List A) — > List A 
flatten nothing = [] 
flatten (just xs) — xs 

apply : {A B : Set} Maybe (A -> List B) A List B 
apply nothing x = [] 
apply (just/) x = fx 

bind : [A B : Set] 

Maybe (List A) Maybe (A List B) List B 
bind xs nothing = [] 
bind xs (just/) = bindi (flatten xs) f 

Figure 2. Helper functions used in the type signature of _»=_. 
Note that there is a reason for not defining bind using the equation 
bind xs f = bindi (flatten xs) (apply /); see Section 4.6. 



data : V {R xs] 

R — > Parser R xs — > List Tok — > Set\ where 

return : x e return x ■ [] 

token : t e token • [ t ] 

|-left : x e p\ ■ s — > x e p\ \ p2 ■ s 

|-right : x e p2 ■ s — > x e Pi \ P2 ■ s 

_»=_ : x e \r pi ■ si — > y e \r (j>2 x) ■ S2 — > 

y e Pi »= P2 ■ *l -H- s 2 
nonempty : x e p ■ t :: .v — > x e nonempty p ■ t :: s 
cast : x e p • .s — > x e cast eqp ■ s 

Figure 3. The semantics of the parser combinators. To avoid clut- 
ter the declarations of bound variables are omitted in the construc- 
tors' type signatures. 



empty list, are defined in Figure 2; bind is defined in terms of bindi, 
the standard list monad's bind operation. The function oo(_)Parser 
is defined as follows, mutually with Parser: 

oo{_)Parser : [A : Set] — > Maybe A — > 

(R : Set) List R Set\ 
oo( nothing )Parser Rxs — oo (Parser R xs) 
oo( just _ )Parser Rxs — Parser R xs 

(oo works also for Seti .) It is straightforward to define a variant of 
\r for this type. It is not necessary to define jf , though: instead of 
conditionally delaying one can just avoid using nothing. 

Just as in Section 3 two additional constructors are included in 
the definition of Parser: 

nonempty : V {R xs} — > Parser R xs — > Parser R [ ] 
cast : V {R xs± xs 2 ] — > xs± 58 bag xs 2 ~~ > 

Parser R xsi — > Parser R x$2 

Here -^bag- stands for bag equality between lists, equality up to 
permutation of elements; the cast combinator ensures that one can 
replace one representation of a parser's initial bag with another. Bag 
equality is denned in two steps. First list membership is encoded 
inductively as follows: 

data _e_ {A : Set} : A -> List A -> Set where 
here : V [x xs] — > x e x :: xs 
there : V [x y xs} — » y e xs — » y e x :: xs 

Two lists xs and ys are then deemed "bag equal" if, for every 
value x, x is a member of xs as often as it is a member of ys: 

-^bag- : v -» List R -> List R -> Set 
xs ^bag y s = V(l)-*I 6 B x e ys 

Here A B means that there is an invertible function from A to 
B, so A and B must have the same cardinality. 

4.1 Semantics 

The semantics of the parser combinators is defined as a relation 
such that x e p ■ s is inhabited iff x is one of the results of 
parsing the string s using the parser p. This relation is denned in 
Figure 3. Note that values of type x e p ■ s can be seen as parse 
trees. 

The parsers come with two kinds of equivalence. The weaker 
one, language equivalence (-^-X is a direct analogue of the equiv- 
alence used for recognisers in Section 3.4: 

: V {R xsi xs 2 } -» 
Parser R xs\ — > Parser R XS2 — > Set[ 
Pi w P2 = ^ i x s ) ~ * x e Pi ' s x e P2 ' s 



Here A o S means that A and 6 are equivalent: there is a function 
of type A — > B and another function of type B — > A. We immedi- 
ately get that language equivalence is an equivalence relation. 

As mentioned above language equivalence is sometimes too 
weak. We may want to distinguish between grammars which define 
the same language, if they do not agree on the number of ways 
in which a given value can be produced from a given string. To 
make the example given above more concrete, the parser backend 
defined in Section 4.2 returns one result when the empty string is 
parsed using return true (parse tree: return), and two results when 
return true | return true is used (parse trees: | -left return and 
| -right return). Based on this observation two parsers are defined 
to be parser equivalent (_=_) if, for all values and strings, the 
respective sets of parse trees have the same cardinality: 

: V {R xs\ XS2} — > 
Parser R xs\ — > Parser R xs 2 — > Set\ 
pi = p2 = V [x s} — > x e p\ ■ s <-> x e p2 ■ s 

From its definition we immediately get that parser equivalence is 
an equivalence relation. Parser equivalence is strictly stronger than 
language equivalence: the former distinguishes between return 
true and return true | return true, while the latter is idempotent. 
Just as in Section 3.2 the initial bag index is correct: 

index-correct : V {R xs x} (p : Parser R xs} — > 
x e p ■ [ ] <-> x e xs 

Note the use of _<->_: the number of parse trees for x matches 
the number of occurrences of x in the list xs. One direction of 
the inverse can be defined by recursion on the structure of the 
semantics, and the other by recursion on the structure of _e_. 

From index-correct we easily get that parsers which are parser 
equivalent have equal initial bags: 

same-bag : V [R xs\ x.s'2} 

[pi : Parser R xs\} \p2 : Parser R XS2} — > 
Pi = P2 -> xsi ^bag xs 2 
Similarly, language equivalent parsers have equal initial sets. 

4.2 Backend 

Following Section 3.3 it is easy to implement a derivative operator 
for parsers: 

D : V [R xs} (t : Tok) (p : Parser R xs) -> 
Parser R (D-bag t p) 

The implementation of the function D-bag which computes the 
derivative's initial bag can be seen in Figure 4. Both D and D-bag 
use analogues of the forced? function from Section 3: 



D-bag : V [R xs} - 
D-bag t (return x) 
D-bag t fail 
D-bag t token 
D-bag t (p! | p 2 ) 



Tok — > Parser R xs — > List R 
= [] 
= [] 
= [t] 

= D-bag t pi -H- D-bag t p 2 



D-bag t (nonempty p) = D-bag t p 
D-bag t (cast eq p) — D-bag t p 
D-bag t (pi P2) with forced? pi \ forced?' P2 
. . . I just/ I nothing = bind^ (D-bag t p\)f 
... Ijust/ Ijustxs = bindi (D-bag t p\) f -H- 

bindi xs (k x — > D-bag t (p 2 x)) 
... I nothing | nothing = [] 

... I nothing I just xs — bindi xs (k x — > D-bag t (p2 x)) 

Figure 4. The index function D-bag. Note that its implementation 
falls out almost automatically from the definition of D. 



forced? : V [A R xs m) — > oo( m )Parser R xs — > Maybe A 
forced? [m = m} _ = m 

forced?' : V {A Ri R 2 : Set} {m} {f : R t -> List R 2 ) -> 

((x : R\) — > oo( m )Parser R 2 (fx)) — > Maybe A 
forced?' [m — m} _ — m 

The non-recursive cases of D, along with choice, nonempty and 
cast, are easy: 



D t (return x) 
D t fail 
D t token 
Dt(pi\ p 2 ) 



= fail 

= fail 

= return t 

= Dtp\ \ Dtp 2 



D t (nonempty p) = Dtp 
D t (cast eq p) —Dtp 

The last case, _»=_, is more interesting. It makes use of the 
combinator return*, which can return any element of its argument 
list: 

return* : V {R} (xs : List R) — > Parser R xs 

return-k [ ] = fail 

return* (x :: xs) — return x | return* xs 

The code is very similar to the code for sequencing in Section 3.3: 

D t (pi »= p 2 ) with forced? p\ \ forced?' p 2 
... I just/ I nothing = D t p\ »= (k x — > (p 2 x)) 
... I nothing | nothing = f D t ( b p{) »= (k x b (p 2 x)) 
... Ijust/ Ijustxi- = D t pi »*= (k x — > p 2 x) 

I return* xs »= (k x — > D t (p 2 x)) 
... I nothing Ijust xs — f D t f 9 p\) p 2 x) 

I return* xs 3= (k x — > D t (p 2 x)) 

There are two main differences. One is the absence of J' . The 
other difference can be seen in the last two cases, where p\ is 
potentially mailable (it is if xs is nonempty). The corresponding 
right-hand sides are implemented as choices, as before. However, 
the right choices are a bit more involved than in Section 3.3. They 
correspond to the cases where p\ succeeds without consuming any 
input, returning one of the elements of its initial bag xs. In this 
case the elements of the initial bag index of p\ are returned using 
return*, and then combined with p 2 using bind. 

The implementation of D-bag is structurally recursive, while the 
implementation of D uses a lexicographic combination of guarded 
corecursion and structural recursion, just as in Section 3.3. It is 
straightforward to prove the following correctness property: 



D-correct : V {R xs x s 1} [p : Parser R xs} — > 
x e D t p ■ s <-> x e p ■ t :: s 

Both directions of the inverse can be defined by recursion on the 
structure of the semantics, with the help of index-correct. 

Given the derivative operator it is easy to define the parser 
backend: 

parse : V {R xs} —> Parser R xs — > List Tok — > List R 

parse [xs = xs] p [] = xs 

parse p (t :: s) = parse (D t p) s 

The correctness of this implementation follows easily from index- 
correct and D-correct: 

parse-correct : V [R xs x s} {p : Parser R xs} — > 
x e p ■ s <-> x e parse p s 

Both directions of the inverse can be defined by recursion on the 
structure of the input string. Note that this proof establishes that 
a parser can only return a finite number of results for a given 
input string (because the list returned by parse is finite) — infinitely 
ambiguous grammars cannot be represented in this framework. 
As mentioned in Section 4.1 we have 

parse (return true | return true) [] = true :: true :: [] . 

It might seem reasonable for parse to remove duplicates from the 
list of results. However, the result type is not guaranteed to come 
with decidable equality (consider functions, for instance), so such 
filtering is left to the user of parse. 

The code above is not optimised, and mainly serves to illustrate 
that it is possible to implement a Parser backend which guarantees 
termination. It is not too hard to see that, in the worst case, parse 
is at least exponential in the size of the input string. Consider the 
following parser: 

p : Parser Bool [ ] 

p = fail 3= k (b : Bool) -> fail 

The derivative D t p is p \ p, for any token t . After taking n deriva- 
tives we get a parser with 2" — 1 choices, and all these choices have 
to be traversed to compute the parser's initial bag. The parser p may 
seem contrived, but similar parsers can easily arise as the result of 
taking the derivative of more useful parsers. 

It may be possible to implement more efficient backends. For in- 
stance, one can make use of algebraic laws like fail 3= p = fail 
(see Section 4.4) to simplify parsers, and perhaps avoid the kind of 
behaviour described above, at least for certain classes of parsers. 
Exploring such optimisations is left for future work, though. 

4.3 Coinductive equivalences 

In Section 3.4 a coinductive characterisation of recogniser equiva- 
lence is given. This is possible also for parser equivalence: 



data _= c - {R xs\ xs 2 } (p\ : 
(P2 ■ 

_::_ : xs\ =a bag xs 2 -> 
(V t -> 00 (D t pi 
PI =c P2 



Parser R xs\) 
Parser R xs 2 ) 

=c Dtp 2 )) -> 



Set where 



Two parsers are equivalent if their initial bags are equal, and, 
for every token t, the respective derivatives with respect to t are 
equivalent (coinductively). Using index-correct and D-correct it is 
easy to show that the two definitions of parser equivalence, 
and _= C -, are equivalent. 

By replacing the use of in the definition of bag equality 
with _o_ we get set equality instead. If, in turn, the use of bag 
equality is replaced by set equality in _= C -, then we get a coinduc- 
tive characterisation of language equivalence (_*»_). 



By using the coinductive characterisations of equivalence I have 
proved that all primitive parser combinators preserve both language 
and parser equivalence, i.e. the equivalences are congruences. 

4.4 Laws 

Let us now discuss the equational theory of the parser combinators. 
Many of the laws from Section 3.4 can be generalised to the setting 
of parser combinators. To start with we have a commutative monoid 
formed by fail and _|_: 

Pl\P2 = P2 I Pi 

fail \p = p 

(Pi I Pi) \P3 = Pi I (P2 I P3) 
If language equivalence is used this monoid is also idempotent: 

P \P ^ P 

We also have a monad, with fail as a left and right zero of bind, and 
bind distributing from the left and right over choice: 

return x 3= p = p x 

p 3= return = p 

pi 3= (X x -> p 2 x 3= p 3 ) = (pi 3= pi) »= p 3 

fail 3= p = fail 

p 3= (A _ — > fail) = fail 

p\ ^ (Xx^> p 2 x \ p 3 x) = pi 3= P2 I PI ^= P3 

(Pi I P2) »= P3 = Pi »= P3 I P2 »*= P3 

Unlike in Section 3.4 there is no need to define a special variant 
of _3=_ to state the laws above: if the types of the argument parsers 
are given (as for »=-left-identity below), then Agda automatically 
infers that bind's implicit arguments xs and / should have the form 
just something. 

Analogues of most of the laws from Section 3.4 are listed 
above. However, assuming that the token type is inhabited, it is 
not possible to find a function 

/ : V {R xs] -> Parser R xs -> List (List R) 
and a Kleene-star-like combinator 

-•k : V [R xs} (p : Parser R xs) — > Parser (List R) (f p) 
such that 

return [] | (p 3= X x ^ p ~k s= X xs — > return (x :: xs)) 
< P* 

holds for all p. (Here is defined as in Section 3.4.) The reason 
is that p may be nullable, in which case the inequality above implies 
that xs e p if • [] must be satisfied for infinitely many lists xs, 
whereas parse-correct shows that a parser can only return a finite 
number of results. (A combinator * satisfying the inequality 
above can easily be implemented if it is restricted to non-nullable 
argument parsers.) 

Before leaving the subject of equational laws, let me take a 
moment to explain how one of the laws above — the left identity 
law for bind — can be proved. Assume that we have already proved 
some of the other laws, along with the following property of bindi: 

bind^-left-identity : 

{A B : Set] (x : A) (f : A -> List B) -> 
bind L [x]f =a bag fx 

I have found the coinductive characterisations of the equivalences 
to be convenient to work with, so I have proved the law roughly as 
follows: 



'»=-left-identity : 

[Rl R 2 : Set} {/ : i?i -> List R 2 ] 
(x : R]) (p : (x : Rt) -> Parser R 2 (fx)) -> 
return* 3= p = c p x 
^S3=-left-identity \f — f] x p — 
bind^-left-identity xf :: X t — > tt ( 

D t (return x 3= p) = c 

fail 3= p | return* [ x ] 3= (X x — > D t (p x)) = c 

fail | return x 3= (X x — > D t (p x)) = c 

return x 3= (X x — > D t (p x)) = c 

Dt(px) □) 

(To avoid clutter the proof above uses the equational reason- 
ing notation ... = c ... = c ... □, and the sub-proofs for the 
individual steps have been omitted.) The proof has two parts. 
First bindi-left-identity is used to show that the initial bags of 
return x 3= p and p x are equal, and then it is shown, for every 
token t, that D t (return x 3= p) and D t (p x) are equivalent. The 
first step of the latter part uses a law relating D and _3=_, the sec- 
ond step uses the left zero law (fail 3= p = c fail) and the right 
identity law for choice (p | fail = c p), the third step uses the left 
identity law for choice (fail | p = c p), and the last step uses the 
coinductive hypothesis. 

The proof as written above would not be accepted by Agda, 
because the coinductive hypothesis is not guarded by constructors 
(due to the uses of transitivity implicit in the equational reasoning 
notation). However, this issue can be addressed (Danielsson 2010). 
For details of how all the properties above have been proved, see 
the code accompanying the paper. 

4.5 Expressive strength 

This subsection is concerned with the parser combinators' ex- 
pressiveness. By using bind one can strengthen the result from 
Section 3.5 to arbitrary sets of tokens: every function of type 
List Tok — > List R can be realised as a parser (if bag equality 
is used for the lists of results). The grammar is similar to the con- 
struction in Section 3.5: 

grammar : V {R} (f : List Tok -> List R) -> Parser R (f [ ]) 
grammar f = token 3= (X t — > It grammar (f o _::_ t)) 
| return* (f [ ]) 

The function grammar satisfies the following correctness property: 

grammar-correct : V {R x s] (f : List Tok — > List R) — > 
x e grammar f ■ s <-> x e / .v 

One direction of the inverse can be defined by induction on the 
structure of the semantics, and the other by induction on the struc- 
ture of the input string. If we combine this result with parse-correct 
we get the expressiveness result: 

maximally-expressive : V {R} (f : List Tok — > List R) {s} — > 
parse (grammar f) s ^bag / s 

Assume for a moment that the primitive parser combinators in- 
cluded sat and applicative functor application (McBride and Pater- 
son 2008) instead of token and bind. Then, for finite sets of tokens, 
we could have defined grammar roughly as in Section 3.5. This 
means that, for finite sets of tokens, the inclusion of the monadic 
bind combinator does not provide any expressive advantage; the 
applicative functor interface is already sufficiently expressive. This 
comparison does not take efficiency into account, though. 

4.6 Examples 

Finally let us consider some examples, along with some practical 
remarks. 



Let us start with the left recursive grammar in the introduction. 
Note that it does not require any user annotations, except for the 
three uses of Agda infers all the type signatures and all the im- 
plicit arguments, including several functions, automatically. Agda's 
inference mechanism is based on unification (a variant of pattern 
unification (Pfenning 1991)), and an omitted piece of code is only 
"filled in" if it can be uniquely determined from the constraints pro- 
vided by the rest of the code. In general there is no guarantee that 
implicit arguments can be omitted, and it is not uncommon that the 
exact form of a definition affects how much can be inferred. 

Consider the definition of bind in Figure 2. It is set up so that 
bind xs nothing evaluates to the empty list, even if xs is a neutral 
term. If bind had instead been defined by the equation 

bindxsf = bindi (flatten xs) (apply f), 

then the example in the introduction would have required man- 
ual annotations: the example gives rise to the constraint xs = 
bind (just xs) nothing, which with the alternative definition of 
bind reduces to xs — bindi xs (X _—>[]), and Agda cannot solve 
this unification problem. 

As an example of a definition for which the initial bag is not 
inferred automatically, consider the following definition of sat: 

sat : V [R] -> (Tok -> Maybe R) -> Parser R _ 
sat {R = R} p = token »= X t — > ok (p t) 
where 

ok-bag : Maybe R — > List R 
ok-bag nothing = _ 
ok-bag (just x) — _ 

ok : (x : Maybe R) — > Parser R (ok-bag x) 

ok nothing = fail 

ok (just x) = return x 

The parser sat p matches a single token t iff p t evaluates to just x, 
for some x; the value returned is x. The initial bag function ok-bag 
is not inferred by Agda. However, the right-hand sides of ok-bag, 
and the initial bag of sat, are inferred. 

The example in the introduction uses the derived combinators 
tok and number. The parser tok, which accepts a given token, is 
easy to define using sat (assuming that equality of tokens can be 
decided using the function _==_): 

tok : Tok — > Parser Tok _ 

tok t = sat (1 t 1 — > if t —— t 1 then just t 1 else nothing) 

Given a parser for digits (which is easy to define using sat) the 
parser number, which accepts an arbitrary non-negative number, 
can also be defined: 

number : Parser N _ 

number — digit + »= return ofoldl (In d — > 10 * n + d) 0 

Here foldl is a left fold for lists, and p + parses one or more ps (as 
in Section 3.1). 

The examples above are quite small; larger examples can also be 
constructed. For instance, Danielsson and Norell (2009) construct 
mixfix operator parsers using a parser combinator library which is 
based on some of the ideas described here. 

5. Conclusions 

A parser combinator library which handles left recursion and guar- 
antees termination of parsing has been presented, and it has been 
established that the library is sufficiently expressive: every finitely 
ambiguous parser on finite input strings which can be implemented 
using the host language can also be realised using the combinators. 

I believe that the precise treatment of induction and coinduction 
which underlies the definition of the parser combinators gives a 



good framework for understanding lazy programs. To take one 
example, Claessen (2004) defines the following parser data type 
using Haskell: 

data P' s a = SymbolBind (s -> P' s a) 
| Fail 

| ReturnPlus a (P' s a) 

He notes that it is isomorphic to the stream processor type used in 
Fudgets (Carlsson and Hallgren 1998), and that this isomorphism 
"inspired the view of the parser combinators being parsing process 
combinators". However, in a total setting I would define these two 
types differently. The stream processors were defined in Section 2, 
with an inductive get constructor and a coinductive put constructor. 
I find it natural to define P' in the opposite way: 

data P' (S A : Set) : Set where 

symbolBind : (5 oo (P 1 S A)) P' S A 

fail : P' SA 

returnPlus : A^P'SA^P'SA 

The reason for the difference is that the types are used differently. 
Stream processors are interpreted using [_], and parsers using 
parse' , which works with finite lists: 

parse' : V {5 A} P' S A List S List (A x List S) 
parse' (symbolBind /) (c :: s) = parse' (f c)) s 
parse' (returnPlus x p) s = (x, a-) :: parse' p s 
parse' _ = [] 

The definition of [_] in Section 2 would not be total if get were 
coinductive, because then we could not guarantee that the result- 
ing colist would be productive. On the other hand, if returnPlus 
were coinductive and symbolBind inductive, then parsers like the 
one used in the proof of maximal expressiveness in Section 4.5 
could not be implemented (consider the case when the argument to 
grammar is X _ — > []). 

The use of lazy data types and general recursion in Haskell is 
very flexible — for instance, Carlsson and Hallgren (1998) use their 
stream processors in ways which would not be accepted if the type 
SP were used in Agda — but I find it easier to understand how and 
why programs work when induction and coinduction are separated 
as in this paper. The use of mixed induction and coinduction has 
been known for a long time (Park 1980), but does not seem to be 
well-known among functional programmers. It is my hope that this 
paper provides a compelling example of the use of this technique. 
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